Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 2 EOL Roadmap? #403

Closed
fake-name opened this issue Jan 23, 2018 · 32 comments
Closed

Python 2 EOL Roadmap? #403

fake-name opened this issue Jan 23, 2018 · 32 comments
Assignees
Labels

Comments

@fake-name
Copy link

LinuxCNC includes a lot of python components, and as everyone is probably aware, Python 2 is EOL in a little more then 2 years.

What's the roadmap for the python 3 migration?

It might be a decent idea to start requiring that people who touch python files make them support either platform. It's completely possible to write python code that works in both versions.

@jepler jepler closed this as completed Jan 23, 2018
@jepler jepler reopened this Jan 23, 2018
@SebKuzminsky
Copy link
Collaborator

I would love an effort to make our python code 2/3 compatible.

Are there tools in the python world that can identify 2/3 compatibility problems in python programs and modules? Maybe through static analysis? "Flake" maybe?

If so we should add build-time tests to run them; for now the test will just warn on failures, then once we have full 2/3 compatibility we'll switch it to fail the build on failures.

@SebKuzminsky
Copy link
Collaborator

pylint --py3k seems like an option. Here's the output for axis:
pylint-axis.txt

@gmoccapy
Copy link
Collaborator

I agree, that we should go the way to write code for both python versions. A good way to start is also the page:

Python future

I will check also on pylint

Norbert

@KurtJacobson
Copy link
Contributor

The python code will be easy to convert to work with both Python 2 and 3, the part I am not so sure about is the C/C++ python bindings. I think boost python is used for those. We probably will have to add a switch to compile for one or the other version of python. Ideally the bindings would work with both, but I am not sure how feasible that is.

@fake-name
Copy link
Author

fake-name commented Jan 23, 2018

I would love an effort to make our python code 2/3 compatible.

Note that you don't have to have the same code work for both. There are many packages written for Py2k that support py3k via 2to3, which is invoked during package installation. Basically, they're written with a little care so that the automatic translation allows them to be converted to python3 without issue.

@c-morley
Copy link
Collaborator

@fake-name as Kurt mentioned, the key problem is the c/c++ binding code.
It seems that we really need to make a choice at one point and change everything all at once.
And python touches almost everything in linuxcnc! Huge job.
Nice job there Python :(

@SebKuzminsky
Copy link
Collaborator

Here's somebody else's experience converting an old, big python2 codebase to python3: https://medium.com/@boxed/moving-a-large-and-old-codebase-to-python3-33a5a13f8c99

There might be something helpful for us to glean from this....

@fake-name
Copy link
Author

fake-name commented Feb 21, 2018

FWIW, I'm rewriting axis.py for work things. Primarily, I'm doing cleanup (holy cow it's the worst python I've ever worked on), but it should be basically 2/3 compatible when I'm done.

It actually resulted in https://github.com/fake-name/autoflake/blob/master/autostar.py, (pr here), which is a kind of cool thing that tries very hard to fix all the from x import * mess in a python file, by doing some crazy ast hijinks to figure out which packages items not defined on the local scope come from, and rename them to their actual full names.

Actually, I'm unclear why axis.py involves tcl at all, but that's a different discussion.

@c-morley
Copy link
Collaborator

SebKuzminsky: An interesting read there. I see nothing of python extensions or python embedding unfortunately, which of course is the hardest part. It's a good reference though.

@jepler
Copy link
Member

jepler commented Feb 21, 2018

@fake-name It's great that you are taking an interest in working on axis and on the externally imposed python2 time bomb that our project faces. Thank you.

But even as you have the urge to rewrite axis.py wholesale (who doesn't? It's more a question of having the energy to do so), please keep in mind how you'll be able to ultimately create that as a pull request where the individual commits are reviewable. This is particularly important since we don't have any tests of GUI components of LinuxCNC, like AXIS. (Unless you're going to fix that too. 😉) A single commit in which the whole structure and design of axis.py is upended is probably unreviewable and is likely to languish indefinitely as a pull request.

The way lots of direct access to Tcl/Tk is mixed with Python/Tkinter does have an historical motivation: Long ago at my $DAY_JOB, I created a GUI builder for Tcl, which I got permission to release as an open source product. (it didn't gain any users and the open source version is unmaintained) A never-realized goal was for programmers to be able to reconfigure the AXIS UI by modifying it in the GUI builder. AXIS never achieved this goal, and eventually the GUI builder files were renamed to ".tcl" and when it felt expedient, developers (mostly me) wrote code in Tcl instead of in Python. Coming from my particular $DAY_JOB's background, mixing Python and Tcl like this doesn't even feel weird; but I know it's not common practice in widely published or read open source Python projects.

@fake-name
Copy link
Author

fake-name commented Feb 21, 2018

But even as you have the urge to rewrite axis.py wholesale (who doesn't? It's more a question of having the energy to do so), please keep in mind how you'll be able to ultimately create that as a pull request where the individual commits are reviewable.

My primary focus here is to get axis into a state where I can extend it as an internal testing tool for work-related things. The rework is basically only because the code is impenetrable as-it-is, due to the complete lack of structure.

Anyways, this would never be a single commit, but rather a series of incremental changes which maintain functionality with incremental refinement.

@rene-dev
Copy link
Collaborator

rene-dev commented Mar 6, 2018

I had a go at this today, and after some help from @jepler stuff started to build.
There is a problem with a dependency for halcompile:
http://theory.stanford.edu/~amitp/yapps/
there is a python 3 compatible version, but its not available in Debian.
I will create a pull request later with what I have done so far.

@rene-dev
Copy link
Collaborator

rene-dev commented Mar 7, 2018

my work in progress branch is here: #416

@loganpowell
Copy link

loganpowell commented Jan 9, 2019

@c-morley
Copy link
Collaborator

c-morley commented Jan 9, 2019

I'll add this proof-of-concept branch of using cython for HAL bindings.
If we also had py3 binding for the motion controller, then screens could use python 3.
Then remap would probably be the next big problem.

https://github.com/LinuxCNC/linuxcnc/tree/python3_cython

@rene-dev
Copy link
Collaborator

I pulled a few python3 fixes into master, and configure and and python syntax checks are now working.
Next step is to fix the python bindings. anyone want to try?
./configure --with-python=python3 --with-boost-python=boost_python-py35

@rene-dev
Copy link
Collaborator

It does compile now, and:
Runtest: 227 tests run, 183 successful, 44 failed + 0 expected
most of the fails are due to the hal python module not working.
the buffer protocol stuff changed, and hal is currently only a dummy module that requires some more time. No gui works. There are a lot of type and import problems.
I am not sure it will be possible to support 2 and 3, but I dont think we need to.
python2 will be EOL in January, and python3 is already 11 years old, and every modern OS comes with it. But I will look into using the preprocessor to make the modules 2 and 3 compatible.

@gibsonc
Copy link

gibsonc commented Sep 24, 2019

Adding my small contribution for those keen on experimenting / assisting

I have been keen to learn C++ for a while and I can find my way around Python so I decided to give this a bash and see how far I get. This is also the first time I have used git or contributed to any project, so please excuse any ignorance.

I have forked 2.8 to https://github.com/gibsonc/linuxcnc/tree/2.8
I have made no attempt to support Python 2 and 3 - it simply spirals out of control with all the if / else constructs. It all compiles now except for halmodule.cc due to the depreciated buffer protocol.

https://docs.python.org/3/c-api/objbuffer.html

I still need to get my head around that. As a result a lot of the tests currently fail.

On Mint 19
pip3 install yapps
./configure --with-python=python3 --with-boost-python=boost_python-py36

@gibsonc
Copy link

gibsonc commented Oct 28, 2019

Just an update for the curious. It is a lot of work, mostly repetitive, but more things are starting to work now. Thanks to @jepler for his assistance. I cut down on another ten failed tests this evening and with what he taught me I think I will be able to resolve a few more quite soon. I am no coder but I am learning fast and if I can learn so can anyone else :)
The only thing that worries me is how we will merge again, but that is a problem for later down the line.

@strufkin
Copy link

strufkin commented Nov 1, 2019

if you interested in i've ported C code and all features i needed worked for me (it was some time ago, hoping this helps you, unfortunately it is a bit behind current master, let me know if i could help here, the only thing i've disabled was related to shmbuffer_procs also found in old mail lists that it is not being used by anyone, also there are no tests for that interface, so seem it needs some discussion )
https://github.com/strufkin/linuxcnc/commits/py3-work

@gibsonc
Copy link

gibsonc commented Nov 1, 2019

Oh man - how I wish I had seen your work before I started. Nothing is wasted however - I have learned a lot from practically nothing. I didn't know about py3c - it makes all this Bytes / Unicode pain go away that I am currently experiencing. I fix it, but it seems to be by chance and that is never a good thing. Ideally the code must check for the types and return the correct type.
It would seem a few of us are duplicating effort in different areas. How can we better pool our efforts and move forward?

@saxa
Copy link

saxa commented Nov 1, 2019

Make a paralel branch and commit faster your work to it :)

@andypugh
Copy link
Collaborator

andypugh commented Nov 2, 2019

Maybe we should all choose one branch and all work on it?

Maybe this one, as it is already in the LinuxCNC repository?
https://github.com/LinuxCNC/linuxcnc/blob/dgarr/py3/PY3_README
But that is only a suggestion.

@spiderdab
Copy link

Hi, I just installed Debian Buster on a VM and compiled dgarr/py3 branch as RIP. No big issues to install. the Buster I downloaded was the '10.2 amd64 net install' iso, and python 2.7 was the default version, so I had to 'update-alternatives' to set python 3.7 as the default version.
I was looking to the python3 porting of lcnc, and came accross this repo which seems to be the more advanced in the migration. I personally needed only hal to be python3 compatible, but if I can I would like to help here with the code (if I'm able to..) or at least doing tests.
Now it is installed in a VM because I'm developing a GUI in GTK3 which uses hal to setup some stepper generators on a mesa 7i96 (..to be changed after for more generators..) and will transfer that on a working machine after it works.

I can report that importing hal works with python3 and also axis seems to work (sim mode only..)

@rene-dev rene-dev self-assigned this Apr 24, 2020
@rene-dev
Copy link
Collaborator

I am currently working on this, picking the best of the attempts that have been made, and getting them up to speed with master.

@rene-dev
Copy link
Collaborator

status update:
Im picking the best parts of the branches that have already been started by other people, and updating them up to current master.

working:

  • Axis
  • All but a few remap related tests pass

not working:

  • Any gtk2 UI, gmoccapy, gscreen, touchy, ...

not tested:

  • qt5
  • actually running a machine

This is the roadmap I came up with:

py3c is not available, on wheezy, so either drop support for wheezy(#743), or see if it still works for python2 if you bundle py3c.

currently the roadblock is gtk2. not all gtk2 bindings are available in python3.
This is also an issue on buster, see #809
I started porting gscreen to gtk3, but the problem is that you cant reliably create a opengl context in gtk3:
https://gitlab.gnome.org/GNOME/gtk/issues/64
https://gitlab.freedesktop.org/mesa/mesa/issues/113

@rene-dev
Copy link
Collaborator

rene-dev commented May 1, 2020

gtk3 gtkglarea problem has been fixed, waiting for them to accept the PR
https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/1790

@rene-dev
Copy link
Collaborator

rene-dev commented May 3, 2020

done, master now build with python3, and all but one tests pass.
I will open individual issues for the new problems.
Thanks to @dngarrett and @strufkin for their work on the python modules, my work is partly based on their work.
Thanks to @jepler for helping with the build system and various python problems.

@rene-dev rene-dev closed this as completed May 3, 2020
@zultron
Copy link
Contributor

zultron commented May 14, 2020

Huge congrats to @rene-dev and the others who accomplished this, and huge thanks, too: we're borrowing from your work over at machinekit/machinekit-hal#114 . You all rock.

@jallwine
Copy link

jallwine commented Oct 2, 2020

When I download the Raspberry Pi 4 image, the linuxcnc python module seems to only be available when running python 2. Does that image not have the latest, or has that not been addressed yet?

@andypugh
Copy link
Collaborator

andypugh commented Oct 2, 2020

LinuxCNC 2.8 still uses Python 2. Master / 2.9 has made the switch to Python 3, but not everything is working yet.

@jallwine
Copy link

jallwine commented Oct 2, 2020

Gotcha, thanks @andypugh!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests